
United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 

Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 223 1 3- 1 450 
www.uspto.gov 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET NO. 



CONFIRMATION NO. 



09/830,899 



08/13/2001 



21003 7590 04/13/2006 

BAKER & BOTTS 

30 rockefeller plaza 

new YORK, NY 10112 



Seungup Pack 



A32095-PCTUS 



5340 



EXAMINER 



LEROUX, ETIENNE PIERRE 



ART UNIT 



PAPER NUMBER 



2I6I 

DATE MAILED: 04/13/2006 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 10/03) 



Off ICQ Action Sumni3rv 


Application No. 

09/830,899 


Applicant(s) 

PAEK ET AL. 


Examiner 

Etienne P LeRoux 


Art Unit 

2161 





~ r/ie MAILING DATE of this communication appears on the cover sheet with the correspondence address - 



Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 (J.S.C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1)^ Responsive to communication(s) filed on 03 March 2006 . 
2a)n Tiiis action is FINAL. 2b)|^ This action is non-final. 

3) 0 Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under £xpa/te Quayle, 1935 CD. 11, 453 O.G. 213. 
Disposition of Claims 

4) [3 Claim(s) 1-43 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) n Claim(s) is/are allowed. 

6) 13 Claim(s) 1-43 is/are rejected. 
?)□ Claim(s) is/are objected to. 

8) 0 Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) 0 The specification is objected to by the Examiner. 

10) 13 The drawing(s) filed on 13 August 2001 is/are: a)|3 accepted or b)^ objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 

11) 0 The proposed drawing correction filed on is: a)n approved b)n disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) 0 The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§119 and 120 

1 3) 0 Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 1 9(a)-(d) or (f). 

a)nAII b)n Some* 0)0 None of: 

1. D Certified copies of the priority documents have been received. 

2. n Certified copies of the priority documents have been received in Application No. . 

3. n Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) n Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 119(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

1 5) 0 Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 1 20 and/or 1 21 . 

Attachment(s) 

1 ) K Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-41 3) Paper No(s). . 

2) EH Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) C] Notice of Informal Patent Application (PTO-152) 

3) n Information Disclosure Statement(s) (PTO-1449) Paper No(s) _ . 6) □ Other: 
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Prosecution Reopened 



In view of the Appeal Brief filed on March 3, 2006, PROSECUTION IS HEREBY REOPENED. 
New grounds of rejection are set forth below. 

To avoid abandonment of the application, appellant must exercise one of the following two options: 

(1) file a reply under 37 CFR 1.111 (if this Office action is non-final) or a reply under 37 CFR 1.1 13 (if 
this Office action is final); or, 

(2) initiate a new appeal by filing a notice of appeal under 37 CFR 41.31 followed by an appeal brief 
under 37 CFR 41.37. The previously paid notice of appeal fee and appeal brief fee can be applied to the 
new appeal. If, however, the appeal fees set forth in 37 CFR 41.20 have been increased since they were 
previously paid, then appellant must pay the difference between the increased fees and the amount 
previously paid. 

A Supervisory Patent Examiner (SPE) has approved of reopening prosecution by signing below: 



CHARLES RONES 
SUPSRM80RY RMENT EXMKSttER 




Claim Status 



Claims 1-43 are pending. Claims 1-43 are rejected as detailed below. 



Priority 

The later-filed application must be an application for a patent for an invention which is also disclosed in 
the prior application (the parent or original nonprovisional application or provisional application). The 
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disclosure of the invention in the parent application and in the later-filed application must be sufficient 
to comply with the requirements of the first paragraph of 35 U.S.C. 1 12. See Transco Products, Inc, v. 
Performance Confraclmg, Inc., 38 F.3d 551, 32 USPQ2d 1077 (Fed. Cir. 1994). 

The disclosure of the prior-filed application, Application No. 60/107,463, filed November 6, 
1 998 fails to provide adequate support or enablement in the manner provided by the first paragraph of 
35 U.S.C. 1 12 for one or more claims of this application. In Appeal Brief, filed March 3, 2006, 
Appellant stated that Figures 7 and 8 and Page 26 provide support for claim 1. Figures 5 and 6 and page 
26 were included under Rule 26 as of the international filing date, i.e., November 5, 1999, The 
disclosure in provisional application 60/107,463, filed November 6, 1998 is not sufficient comply with 
the requirements of the first paragraph of 35 U.S.C. 1 12 regarding the claimed invention as filed August 
. 13,2001. 

Claim Rejections - 35 USC § 102 
1. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis 
for the rejections under this section made in this Office action: 

The following is a quotafion of the appropriate paragraphs of 35 U.S.C. 102 that form the basis 
for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on sale in this country, 
more than one year prior to the date of apph cation for patent in the United States. 

Claims 1-43 are rejected under 35 U.S.C. 102(b) as being anticipate by US Pat No 6,079 issued 
to Eleftheriadis et al (hereafter Pat '566). 
Claims K 17 and 33: 
Pat '566 discloses: 
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(a) at least one multimedia information input interface receiving said multimedia information [MPEG- 
4 file. Fig 4, electronic memory 390, col 7, line 25] 

(b) a computer processor coupled to said at least one multimedia information input interface receiving 
said multimedia information therefrom [Fig 4, CPU 380], 

processing said multimedia information by performing object extraction processing to generate 
multimedia object descriptions from said multimedia information [read operation module 290 accesses 
object table 370 for translation purposes and communicates extracted audiovisual data to MPEG-4 
player 360, col 7, lines 35-40] 

processing said generated multimedia object descriptions by object hierarchy processing to generate 
muhimedia object hierarchy descriptions indicative of an organization of said object descriptions [col 

3, lines 35-40, tree-structured approach] 

wherein at least one description record including said multimedia object descriptions and said 
multimedia object hierarchy descriptions is generated for content embedded within said multimedia 
information [Fig 4] 

( c ) a data storage system, operatively coupled to said processor for storing at least said at least one 
description record [Fig 4, MPEG-4 player 360, video buffer col 7, lines 40-42] 
Claims 2, 18 and 34: 

Pat '566 discloses wherein said multimedia information comprises image information, said multimedia 
object descriptions comprise image object descriptions, and said multimedia object hierarchy 
descriptions comprise image object hierarchy descriptions, [col 3, lines 35-40, tree-structured 
approach] 
Claims 3 and 19: 

Pat '566 discloses (a) image segmentation processing to segment each image in said image information 
into regions within said image, and (b) feature extraction processing to generate one or more feature 
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descriptions for one or more of said regions, whereby said generated object descriptions comprise said 
one or more feature descriptions for one or more of said regions [col 3, lines 20-28] 
Claims 4. 20 and 35: 

Pat '566 discloses wherein said one or more feature descriptions are selected from the group consisting 
of text annotations, color, texture, shape size and position [col 30-40] 
Claims 5, 21 and 36: 

Pat '566 discloses wherein said object hierarchy processing comprises physical object hierarchy 
organization to generate physical object hierarchy descriptions of said image object descriptions that 
are based on spatial characteristics of said objects, such that said image object hierarchy descriptions 
comprise physical descriptions [col 30-40]. 
Claims 6, 22 and 37: 

Pat '566 discloses wherein said object hierarchy processing further comprises logical object hierarchy 
organization to generate logical object hierarchy descriptions of said image object descriptions that are 
based on semantic characteristics of said objects, such that said image object hierarchy descriptions 
comprise physical and logical descriptions [col 2, lines 5-10] 
Claims 7 and 23: 

Pat '566 discloses image segmentation processing to segment each image in said image information 
into regions within said image and (b) feature extraction processing to generate object descriptions for 
one or more of said region, and wherein said physical hierarchy organization and said logical hierarchy 
generate hierarchy descriptions of said object descriptions for said one or more of said regions [col 2, 
lines 5-10] 
Claims 8 and 24: 

Pat '566 discloses fiarther comprising an encoder receiving said image object hierarchy descriptions 
and said image object descriptions, and encoding said image object hierarchy descriptions and said 
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image object descriptions into encoded descriptions information, wherein said data storage system is 
operative to store said encoded description information as said at least one description record [Fig 4, 
390] 

Claims 9, 25 and 38: 

Pat '566 discloses wherein said multimedia information comprises video information, said multimedia 
object descriptions comprise video object descriptions including both event descriptions and object 
descriptions, and said muhimedia hierarchy descriptions comprise video object hierarchy descriptions 
including both event hierarchy descriptions and object hierarchy descriptions [col 1, lines 30-40] 
Claims 10 and 26: 

Pat '566 discloses (a) temporal video segmentation processing to temporally segment said video 
information into one or more video events or groups of video events and generate event descriptions 
for said video events, (b) video object extraction processing to segment said one or more video events 
or groups of video events into one or more regions, and to generate object descriptions for said regions; 
and (c) feature extraction processing to generate one or more event feature descriptions for said one or 
more video events or groups of video events, and one or more object feature descriptions for said one 
or more regions; wherein said generated video object descriptions include said event feature 
descriptions and said object descriptions [col 3, lines 30-40] 
Claims 1 K 27 and 39: 

Pat '566 discloses wherein said one or more event feature descriptions are selected from the group 
consisting of text annotations, shot transition, camera motion, time and key frame, and wherein said 
one or more object feature descriptions are selected from the group consisting of color, texture, shape, 
size, position, motion, and time [col 3, lines 27, 28, col 3, lines 30-35] 
Claims 12. 28 and 40: 
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Pat '566 discloses wherein said object hierarchy processing comprises physical event hierarchy 
organization to generate physical event hierarchy descriptions of said video object descriptions that are 
based on temporal characteristics of said video objects, such that said video hierarchy descriptions 
comprise temporal descriptions [col 3, lines 15-45] 
Claims 13, 29, 41 and 43: 

Pat '566 discloses wherein said object hierarchy processing further comprises logical event hierarchy 
organization to generate logical event hierarchy descriptions of said video object descriptions that are 
based on semantic characteristics of said video objects, such that said hierarchy descriptions comprise 
both temporal and logical descriptions [col 4, lines 1-15] 
Claims 14, 30 and 42: 

Pat '566 discloses wherein said object hierarchy processing further comprises physical and logical 
object hierarchy extraction processing, receiving said temporal and logical descriptions and generating 
object hierarchy descriptions for video objects embedded within said video information, such that said 
video hierarchy descriptions comprise temporal and logical event and object descriptions [col 3, line 
15 through col 4, line 15] 
Claims 15 and 3 1 : 

Pat '566 discloses (a) temporal video segmentation processing to temporally segment said video 
information into one or more video events or groups of video events and generate event descriptions 
for said video events, (b) video object extraction processing to segment said one or more video events 
or groups of video events into one or more regions, and to generate object descriptions for said 
regions', and (c) feature extraction processing to generate one or more event feature descriptions for 
said one or more video events or groups of video events, and one or more object feature descriptions 
for said one or more regions; wherein said generated video object descriptions include said event 
feature descriptions and said object descriptions, and wherein said physical event hierarchy 
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organization and said logical event hierarchy organization generate hierarchy descriptions from said 
event feature descriptions, and wherein said physical object hierarchy organization and said logical 
object hierarchy organization generate hierarchy descriptions from said object feature descriptions [col 
3, line 15 through col 4, line 15] 
Claims 16 and 32: 

Pat '566 discloses an encoder receiving said video object hierarchy descriptions and said video object 
descriptions, and encoding said video object hierarchy descriptions and said video object descriptions 
into encoded description information, wherein said data storage system is operative to store said 
encoded description information as said at least one description record [col 3, line 15 through col 4, 
line 15] 

Response to Arguments 

Applicant's arguments filed in Appeal Brief of 3/3/2006, have been fully considered and are 
partially persuasive. Applicant's arguments are now moot based on above new art rejection. For 
purposes of clarity of the record, certain arguments made by Appellant are considered below. 
Appellant Argues: 

Appellant in pages 8 and 9 argues that instant invention pertains to the MPEG-7 standard while 
Rajan's disclosure is with reference to MPEG-4. 
Examiner Responds: 

Examiner is not persuaded. In response to applicant's argument that the references fail to show 
certain features of applicant's invention, it is noted that the features upon which applicant relies (i.e., 
M PEG-7) are not recited in the rejected claim(s). Although the claims are interpreted in light of the 
specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 
F.2d 1181, 26USPQ2d 1057 (Fed. Cir. 1993). 
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Appellant Argues: 

Appellant states in the paragraph joining pages 9 and 10: 
Rajan does not disclose or suggest at least the claimed "performing object extraction processing to 
generate multimedia object descriptions." Indeed, the lack of such disclosure in Rajan is not surprising 
since although "object extraction processing to generate multimedia object descriptions" is a key step 
in the present invention (in order to extract information from a multimedia signal, such as a picture or 
video, to describe the content of the picture or video), it is entirely unnecessary for the purposes of 
MPEP-4 and Rajan (which are directed to multimedia composition and presentation, and not 
extraction). 
Examiner Responds: 

Examiner is not persuaded. In response to applicant's argument that the references fail to show 
certain features of applicant's invention, it is noted that the features upon which applicant relies (i.e., 
unnecessary for the purposes of MPEG-4) are not recited in the rejected claim(s). Although the claims 
are interpreted in light of the specification, limitations from the specification are not read into the 
claims. Sec In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir 1993) 

Appellant in Summary of the Invention in the Appeal Brief filed March 3, 2006 points to Figure 
7 of the drawings and to paragraph 3 of page 26 of the specification which states: 
In a preferred method, object extraction 720 consists of two subsidiary operations, namely image 
segmentation 725 and feature extraction and annotation 726. For the image segmentation 725 step, 
any (emphasis added) region tracking technique which partitions digital images into regions that share 
one or more common characteristics may be employed. Likewise, for the feature extraction and 
annotation step 326, any (emphasis added) technique which generates features from segmented regions 
may be employed. 
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The specification slates that any region tracking technique and any technique that generates 
features from segmented regions may be employed to perform object extraction. Examiner concludes 
that the specification is stating that "performing object extraction processing to generate multimedia 
object descriptions" would have been obvious to one of ordinary skill in the art because no particular 
process is described for performing object extraction. Because no particular process for object 
extraction is described, applicant is admitting that "performing object extraction processing to generate 
multimedia object descriptions" is well-known and expected in the art and is thus admitted prior art. 
Appellant Argues: 

Appellant states in the paragraph joining pages 16 and 17: 
Additionally, though it apparently did not form the basis for any rejections, the Examiner relies on a 
1999 MPEG-4 publication as alleged evidence of the state of the art at the time the present application 
was filed. (See Final Office Action, p. 9). Hov^ever, the present appUcation claims priority to 
November 6, 1998. For this reason at least, the cited reference is not a valid indicator of the state of 
the art at the time of the present invention. Additionally, as explained in detail above, the cited 
reference refers to the lVIPEG-4 technology, which is distinct from, and unrelated to, the claimed 
invention. 

Exaniiner Responds: 

Examiner is not persuaded. Examiner has repeatedly pointed out in the above 
Response(s) by Examiner that limitations from the specification cannot be read into the claims 
because examiner during the examination process, must give claims their broadest reasonable 
interpretation. Furthermore, as pointed out in above Office action, deficiencies in Appellant's 
provisional apphcation filed November 6, 1998, obviates the priority date of provisional 
application filed November 6, 1998. 
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Furthermore, examiner provides additional support* for the fact that MPEG-4 includes 
object descriptors. In fact, the MPEG-4 and MPEG-4 share common object descriptors. 

Contact Information 

Any inquiry concerning this communication or earlier communications from the examiner should be 
directed to Etienne P. LeRoux whose telephone number is (571) 272-4022. The examiner can normally 
be reached Monday through Friday between 8:00 am and 4:30 pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, 
Jeffrey Gaffin, can reached on (571) 272-4146. The fax phone number for the organization where this 
application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent Application 
Information Retrieval (PAIR) system. Status information for published applications may be obtained 
from either Private PAIR or Public PAIR. Status information for unpublished applications is available 
through Private PAIR only. For more information about the PAIR system, see http://pair- 
direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the 
Electronic Business Center (EBC) at 866-217-9197 (toll-free). 

Etienne LeRoux 
April 10, 2006 



^ Pub No US 2005/0076055, Mory et al, ptiragraph 39 



